Displays

ABSTRACT

A method and apparatus are provided for controlling a display device to generate an image for viewing by a user on a display, the image being formed by one or more light having predetermined relative timings. Received image data define one or more light components of a feature to be displayed as an element in the image. Rate data are received, e.g. from an associated tracker system, indicative of a rate of change in orientation of the display relative to a direction of gaze of an eye of the user. The received rate data are used to determine a position on the display for displaying each of the one or more light components of the feature including determining any respective adjustment required to a determined position for the light component according to the received rate data and the predetermined relative timings. The determined position of each of the one or more light components is output to the display device for display of the light component at the respective determined position.

This invention relates to display methods and devices for generating animage for display to a viewer. In particular, but not exclusively, theinvention provides a method and apparatus for generating images forviewing on displays in such a way as to increase display luminance andto reduce display ‘flicker’ as may arise when there is relative movementof the display and a viewer's direction of gaze. The invention may beapplied in particular to substantially transparent head orhelmet-mounted display (HMD) systems.

It is known to attempt to achieve an increase in brightness of imagesfor display in HMD systems by increasing the proportion (duty cycle) ofan image refresh period, e.g. a period of 16.667 ms in the case of a 60Hz display, during which a pixel of a display device is emitting light.In a digital display system, for example, this means that the pulsewidth modulation (PWM) scheme being used is required to allow for apixel to emit light pulses over a greater proportion of the imagerefresh period. Given limitations in the achievable brightness ofdisplay illumination sources, e.g. light emitting diodes (LEDs), andlimitations in optical efficiency, an increase in total light pulseduration during an image refresh period may be the only option forincreasing image brightness.

It is also known to attempt to reduce display flicker by increasing theduration or to increase the number of light pulses emitted during animage refresh period.

However, it is also known that increasing the total duration over whichlight pulses may be emitted by a pixel in an HMD, for example, bringsunwanted display artefacts when there is relative movement of the HMDand a direction of gaze of the viewer while displaying a generatedimage. In particular, the viewer may perceive a ‘smearing’ of thegenerated image over a region of the display during such relativemovement. This problem is particularly noticeable when the viewer isobserving an external scene against which a so-called ‘space-stabilised’symbol is being displayed; the space-stabilised symbol intended toappear in the display overlain at a fixed position relative to a line ofsight to a point in the external scene, irrespective of head movement.For this reason, it is generally desirable for a pixel to emit its lightover as short a period as possible to reduce the incidence of imagesmearing, but with the associated limitation on maximum achievable pixelbrightness and increased chance of seeing display flicker effects.

According to a first aspect of the present invention, there is provideda method for controlling a display device to generate an image forviewing by a user on a display, the image being formed by two or morelight components having predetermined relative timings, the methodcomprising the steps:

(i) receiving image data defining one or more light components of afeature to be displayed as an element in the image;

(ii) receiving rate data indicative of a rate of change in orientationof the display relative to a direction of gaze of an eye of the user;

(iii) determining a position on the display for displaying each of theone or more light components of the feature including determining anyrespective adjustment required to a determined position for the lightcomponent according to the received rate data and the predeterminedrelative timings; and(iv) outputting the determined position of each of the one or more lightcomponents to the display device for display of the light component atthe respective determined position.

By this method, display brightness may be increased and the effectiverate of displayed image refresh may also be increased without incurringproblems of image smear or multiple image effects during relativemovement of the display and the direction of gaze of a viewer's eye.Relative movement about any of the possible axes, i.e. in azimuth,elevation or roll, may be compensated for when determining the positionat which to display each of the multiple light components of a featurewithin an image, if different to the position at which the feature wouldhave been displayed in the absence of relative movement. In this way,multiple light components may be used within an image refresh period toachieve a required brightness level and each light component will beperceived to be correctly positioned on the display at the time it isdisplayed.

In an example embodiment, one of the two or more light components isselected as a reference light component such that, at step (iii), anysaid respective adjustment is determined using the timing of the lightcomponent relative to a timing of the selected reference lightcomponent, if different. This has the processing advantage in that anadjustment is not required for one of the light components whendetermining its position on the display.

In an example embodiment, the two or more different light componentscomprise three light components to be displayed sequentially and whereinthe selected reference light component is the second light component tobe displayed of the three light components. In the event that the lightcomponents of each component image are equally spaced in time, theadjustment required in respect of the first component has a simplerelationship to the adjustment required for the third component, with acorresponding reduction in processing time.

In a further example embodiment, at step (iii), determining a positionfor displaying the one or more light components comprises receiving dataindicative of an orientation of the display in inertial space anddetermining a position on the display such that the feature appearsaligned to a predetermined line of sight to a point in inertial space.In that example, determining any respective adjustment may comprisedetermining a respective adjustment to the received data indicative ofan orientation of the display such that the position of the lightcomponent is determined after applying the respective adjustment to thereceived orientation data.

In an example embodiment, the rate data measure a rate of change inorientation of the display in inertial space and the measured rate isindicative of the rate of relative movement of the display relative tothe direction of gaze of the user assuming that the direction of gaze ofthe user is substantially fixed in inertial space.

In certain applications of the present invention, the opportunity forsignificant roll movements may be limited, such that the received ratedata may define a rate of change in orientation of the display inazimuth and in elevation resolved in a frame of reference of thedisplay, assuming no change in orientation of the display about a rollaxis in the frame of reference of the display and assuming that thedirection of gaze of the user is substantially fixed in inertial space.This enables only two components of rate data to be considered whenpositioning light components of features on a display. In particular, atstep (iii), determining any respective adjustment comprises determininga linear displacement from a position at which the light component wouldbe displayed if there was no relative movement of the display and thedirection of gaze of the user, comprising a displacement in azimuthacross the display determined using the received rate of change inorientation in azimuth combined with a displacement in elevation acrossthe display determined using the received rate of change in orientationin elevation to give a net linear displacement across the display fordisplay of the light component. The calculation of a linear shiftprovides for a simpler calculation of position than an adjustment to arotation, enabled by the assumption of a zero roll rate.

In an example embodiment, the two or more light components compriselight components displayed in three respective phases, the phases havingsaid predetermined relative timings.

In a further example embodiment, at step (iii), determining a positionon the display for each of the one or more light components of thefeature further comprises taking account of whether the image projectionis a flat surface projection or a spherical projection. This enables amore precise positioning for display of the light components,recognising the effect of the type of image projection on the perceivedposition of space-stabilised symbols.

In an example embodiment, the received rate data are received from, ordetermined from output by, an eye tracker system associated with thedisplay. Such a tracker enables relative movement of the display and thedirection of gaze to be determined whether due to the user maintaining afixed gaze upon an externally visible feature while the display moves,or due to changes in the direction of gaze of the eye, or a combinationof the two.

In an example embodiment, the display is a head or helmet-mounteddisplay (HMD) arranged to display collimated images. In such anapplication, the rate data may be received from, or derived from outputby, a head or helmet tracker system.

According to a second aspect of the present invention there is provided,with equivalent advantages and benefits to the method of the firstaspect, an apparatus for controlling a display device to generate animage for viewing by a user on a display, the image being formed by twoor more light components having predetermined relative timings, theapparatus comprising:

an input for receiving image data defining one or more light componentsof a feature to be displayed as an element in the image;

an input for receiving rate data indicative of a rate of change inorientation of the display relative to a direction of gaze of an eye ofthe user; an image processor arranged:

(i) to determine a position on the display for displaying each of theone or more light components of the feature including determining anyrespective adjustment required to a determined position for the lightcomponent according to the received rate data and the predeterminedrelative timings; and

(ii) to output the determined position of each of the one or more lightcomponents to the display device for display of the light component atthe respective determined position.

In an example embodiment of the apparatus, one of the two or more lightcomponents may be selected as a reference light component such that, atstep (i), the image processor is arranged to determine any saidrespective adjustment using the timing of the light component relativeto a timing of the selected reference light component, if different.

In an example embodiment of the apparatus, the two or more differentlight components comprise three light components to be displayedsequentially and wherein the selected reference light component is thesecond light component to be displayed of the three light components.

In a further example embodiment of the apparatus, the two or more lightcomponents may comprise three light components and the predeterminedrelative timings may define equal time intervals between the times ofdisplay of the three light components.

In an example embodiment of the apparatus, at step (i), the imageprocessor is arranged to determine a position for displaying the one ormore light components by receiving data indicative of an orientation ofthe display in inertial space and determining a position on the displaysuch that the feature appears aligned to a predetermined line of sightto a point in inertial space. In this example, the image processor maybe arranged to determine any respective adjustment by determining arespective adjustment to the received data indicative of an orientationof the display such that the position of the light component isdetermined after applying the respective adjustment to the receivedorientation data.

In a further example embodiment of the apparatus, the received rate datameasure a rate of change in orientation of the display in inertial spaceand the measured rate is indicative of the rate of relative movement ofthe display relative to the direction of gaze of the user assuming thatthe direction of gaze of the user is substantially fixed in inertialspace.

In an example embodiment of the apparatus, the received rate data definea rate of change in orientation of the display in azimuth and inelevation resolved in a frame of reference of the display, assuming nochange in orientation of the display about a roll axis in the frame ofreference of the display and assuming that the direction of gaze of theuser is substantially fixed in inertial space. This assumption enablesthe image processor, at step (i), to determine any respective adjustmentby determining a linear displacement from a position at which the lightcomponent would be displayed if there was no relative movement of thedisplay and the direction of gaze of the user, comprising a displacementin azimuth across the display determined using the received rate ofchange in orientation in azimuth combined with a displacement inelevation across the display determined using the received rate ofchange in orientation in elevation to give a net linear displacementacross the display for display of the light component

As for the first aspect, the two or more light components comprise lightcomponents displayed in three respective phases, the phases having saidpredetermined relative timings.

In a further example embodiment of the apparatus, the image processor isarranged, at step (i), to determine a position on the display for eachof the one or more light components of the feature taking account ofwhether the image projection is a flat surface projection or a sphericalprojection.

The received rate data may be received from, or determined from outputby, an eye tracker system associated with the display.

In a particular application of the second aspect of the presentinvention, the display is a head or helmet-mounted display (HMD)arranged to display collimated images. In that application, the ratedata are received from, or derived from output by, a head or helmettracker system.

According to a third aspect of the present invention, there is provideda head or helmet-mounted display (HMD) system comprising a displaydevice for generating images for display on a head or helmet-mounteddisplay and an image processor arranged to implement the methodaccording to the first aspect of the invention.

According to a fourth aspect of the present invention, there is provideda computer program product comprising a computer-readable medium, ormeans for access thereto, having stored thereon computer program codewhich when loaded onto a digital processor and executed is arranged toimplement the method according to the first aspect of the presentinvention.

According to a fifth aspect of the present invention, there is providedan image processor programmed to implement the method according to thefirst aspect of the present invention.

Example embodiments of the present invention will now be described inmore detail with reference to the accompanying drawings, of which:

FIG. 1a provides a representation of a known method for generating apixel using a sequence light pulses or image components of anappropriate duration and brightness within a given frame period or imagerefresh period;

FIG. 1b provides a representation of the known effect of eye movementrelative to the display upon the perceived position of displayedcomponents of a pixel;

FIG. 2 is a flow diagram showing the steps in a known process forgenerating an image including space-stabilised symbols in a head orhelmet-mounted display (HMD);

FIG. 3 is a flow diagram showing an improved image generation processaccording to example embodiments of the present invention;

FIG. 4a provides an illustration of how the generation of imagecomponents may be varied in a region of an HMD image area by exampleembodiments of the present invention;

FIG. 4b provides an illustration of what a viewer perceives when viewingthe region of an HMD image area subject to the variations shown in FIG.4a according to example embodiments of the present invention; and

FIG. 5 shows the components of a typical HMD system in which exampleembodiments of the present invention may be implemented.

Known head or helmet-mounted display (HMD) systems include a digitaldisplay device, under the control of an image processor, and atransparent combiner in the form of a helmet visor or a waveguidepositioned in front of one or other eye of a user in the user's line ofsight to an external scene so that images output by the display devicemay be projected onto the interior surface of the visor and reflectedtowards the viewer's eye or conveyed through the waveguide and outputalong that line of sight to appear overlain on the user's view of theexternal scene. The display device may comprise a digital micro-mirrordevice (DMD) or Liquid Crystal on Silicon (LCoS) display device, forexample, having pixel-sized elements each separately controllable toreflect, emit or transmit light from one or more illuminating lightsources, according to the type of display device. Light may be outputfrom the display device at each pixel position in the form of apredefined sequential combination of discrete pulses of light of equalintensity but of a variety of durations. The eye integrates the discretelight pulses output at each pixel position over an image refreshperiod—16.667 ms in the case of an example 60 Hz display refreshrate—and perceives a pixel of a brightness determined by the totalduration of pulses output during the image refresh period.

Of course other methods as would be known to a person of ordinary skillin the art are available for achieving a required level of light outputby a pixel in a display during an image refresh period. In the case of acolour display, a corresponding display device would be controlled toemit pulses of red, green and blue light sequentially during an imagerefresh period of a relative duration determined by the colour that aviewer is intended to perceive.

However, as mentioned above, whichever method is used to cause a pixelto emit light of an appropriate duration, the greater that totalduration, the greater are the chances of a viewer perceiving imagesmearing effects, including a perception of multiple images, when thereis relative movement of a user's eye relative to the display image area.Such relative movement may be due either to movement of the eye itselfto alter the direction of gaze to different features in the externalscene (saccadic eye movement), or to movement of the head, and hence ofthe display, while the eye maintains a fixed gaze upon a feature visiblein the external scene (vestibulo-ocular reflex, in the case where thefixed direction of gaze is to fixed feature in inertial space).

Example embodiments of the present invention provide a way to increasethe overall light output by a pixel in a display device and to reducethe incidence of display flicker without incurring the above-mentionedsmearing or multiple image effects in a monochrome display. Theinvention provides a way to control a display device to increase theproportion of an image refresh period during which a pixel may emitlight by dividing the image refresh period into discrete phases or pulseperiods during which ‘light components’ of the image or of symbols orfeatures within the image may be displayed, being either periods ofcontinuous illumination or periods comprising a set of light pulsesaccording to an existing pulse width modulation (PWM) scheme, the PWMscheme being therefore repeated in respect of each light component. Eachlight component is subject to modified treatment according to theinvention to avoid the above-mentioned smearing effects that wouldotherwise continue to occur, as will be described briefly with referenceto FIG. 1.

Referring initially to FIG. 1a , an example division of an image refreshperiod into three phases is shown during which component light Pulses 1,2 and 3 respectively are displayed, in this example of substantiallyequal duration, for Pixel 2 in a group of three adjacent pixels (Pixels1, 2 and 3) within the image area of a display. It is assumed in thisrepresentation that there is no change of eye gaze direction relative tothe display during the periods in which Pulses 1, 2 and 3 and beingemitted so that the viewer perceives Pixel 2 to be a pixel of abrightness represented by the total duration of Pulses 1, 2, and 3.

Referring to FIG. 1b , the effect of eye movement relative to thedisplay can be seen to cause Pulse 2 and Pulse 3 to appear displacedfrom the intended pixel position at the times they are generated. Theresult of this displacement is that the eye integrates differentcombinations of light from Pulses 1, 2 and 3 in the region of thedisplay covering pixels 2 and 3, in this example, and the viewerperceives the light intended for Pixel 2 to be smeared according to theextent of displacement of the three component light pulses over thatregion. Pixel 2 no longer appears to have the intended brightness andthe viewer sees a range of varying light levels resulting from thecombinations of Pulse 1 alone, Pulse 1 and Pulse 2, Pulse 2 and Pulse 3and Pulse 3 alone according to the extent of positional overlap in theperceived displaced positions of the three pulses.

If the three component light pulses shown in FIG. 1 were of differentcolours, then the effect of displacement of light pulses would beperceived as a break-up of the colours intended for Pixel 2, with theviewer seeing a range of different colours according to the overlappingcombinations of pulse colour at each position on the display.

The problem illustrated in FIG. 1b occurs in particular when the displaysystem is displaying so-called ‘space-stabilised’ symbols that areintended to appear in the display as if fixed in space relative to aline of sight to a point in an externally visible scene, irrespective ofhead movement. In order to appear fixed in space, a space-stabilisedsymbol must be continually repositioned in the display, using displayorientation data supplied by an associated tracker system, to compensatefor movement of the viewer's head or helmet and hence of the displayrelative to the viewer's direction of gaze. As the viewer's gaze tendsto remain fixed on a line of sight to a feature visible in inertialspace during head movement, not necessarily the same point as that towhich the symbol is aligned, light pulses generated in displaying therepositioned pixels of space-stabilised symbols may be received atdifferent points on the retina of the viewer's eye, resulting in theperceived effect shown in FIG. 1 b.

According to example embodiments of the present invention, a scheme isprovided for generating images using light pulses emitted duringmultiple discrete phases or pulse periods over a chosen proportion of animage refresh period in such a way as to avoid image smearing ormultiple image effects.

To put the present invention into context, a known method forpositioning symbols or other features in generated images will firstlybe described with reference to FIG. 2 and improvements to that methodprovided by the present invention will then be described with referenceto FIG. 3.

Referring initially to FIG. 2, a flow diagram is provided showing stepsin a known process for generating an image comprising fixed andspace-stabilised symbols for display by an HMD. The process begins atSTEP 10 determining the orientation of the display relative to aninertial frame of reference. This step may be carried out by a head orhelmet tracker system, for example as described in co-pending UK patentapplication GB1516120.1 by the present Applicant, assuming that thedisplay is fixed immovably to the head or helmet. Orientation datadetermined at STEP 10 are used at STEP 15 to calculate the requiredposition in the image area of the display of each space-stabilisedsymbol for a current image refresh period. Each symbol is ‘drawn’ atSTEP 20, generating a data set determining the required brightness ofeach pixel in the image area of the display over the image refreshperiod to display the generated symbols at their determined positions. Adata set defining the drawn symbols is stored in an image store 25.

Any symbols required to be displayed at fixed positions within the imagearea of the display may be positioned and drawn at STEP 30 and theresultant data set stored in the image store 25.

At STEP 35, the contents of the image store 25 are output and used togenerate video data, combining the determined pixel characteristicsdefined at STEP 20 with those defined at STEP 30. The generated videodata are then output to a display device of the display system togenerate the image for the image refresh period such that, at STEP 40,it appears on the display visible to the viewer. The process thenresumes at STEP 10 for the next image refresh period.

In a known technique for calculating, at STEP 15, the position ofsymbols required to appear aligned in an HMD to a line of sight to apoint in inertial space, the orientation of the HMD in inertial space,as determined by an associated tracker system at STEP 10, may berepresented by a rotation matrix [HW],

$\quad\begin{bmatrix}{{\cos(e)}{\cos(a)}} & {{\cos(e)}{\sin(a)}} & {- {\sin(e)}} \\{{{\sin(r)}{\sin(e)}{\cos(a)}} - {{\cos(r)}{\sin(a)}}} & {{{\sin(r)}{\sin(e)}{\sin(a)}} + {{\cos(r)}{\cos(a)}}} & {{\sin(r)}{\cos(e)}} \\{{{\cos(r)}{\sin(e)}{\cos(a)}} + {{\sin(r)}{\sin(a)}}} & {{{\cos(r)}{\sin(e)}{\sin(a)}} - {{\sin(r)}{\cos(a)}}} & {{\cos(r)}{\cos(e)}}\end{bmatrix}$where a, e and rare Euler angles in azimuth, elevation and roll,respectively.

The matrix [HW] defines a transformation of a known line of sight vectorP_(W) to a point in inertial space into a line of sight vector P_(H) tothe point in a frame of reference of the HMD. Thus,P _(H)=[HW]P _(W)  (1)

The vector P_(W) would typically be known from associated systems. Forexample, if the feature were a known way-point, the relative position ofthe user and hence of the display in inertial space would be known fromGPS or other sources. At STEP 15, the determined line of sight vectorP_(H) may be converted into display coordinates, according to:

$\begin{matrix}{X_{D} = {{S\frac{y_{H}}{x_{H}}\mspace{31mu} Y_{D}} = {S\frac{z_{H}}{x_{H}}}}} & (2)\end{matrix}$where (X_(D), Y_(D)) are the display coordinates, X_(H), Y_(H) and Z_(H)are the components of the vector P_(H) and S is a scaling factor for thedisplay.

The inventors in the present invention have realised that the rate ofchange in orientation of the HMD, as would typically be available from atracker system or derivable from data output from orientation dataoutput by a tracker system, may also be input to the calculation ofsymbol positioning so that an adjustment to the matrix [HW] in the formof a rotation matrix [H′H] may be calculated in respect of each lightcomponent, based upon that rate of change in HMD orientation and thetiming of the phase in which a light component is to be displayedrelative to the time of displaying light components in a selected one ofthe other phases, chosen as a reference phase. The adjustment matrix[H′H] may then be used in the calculation of an adjusted line of sightvector P′_(H), modifying equation (1) as follows:P′ _(H)=[H′H][HW]P _(W)  (3)

The adjusted vector P′_(H) applicable to a given light component maythen be used to calculate an adjusted position for the light componenton the display relative to a light component of the selected referencephase. In this calculation, the components of the vector P′_(H) replacethose of the unadjusted vector P_(H) in equations (2) to give thedisplay position of the light component. No adjustment in displayedposition is required for the light component in the selected referencephase whose displayed position remains that calculated according toequations (1) and (2) using the unadjusted matrix [HW].

The calculated positions are used to draw the respective lightcomponents of the symbol such that each light component appears in thedisplay to be aligned to the direction of its respective adjusteddirection P′_(H) or, in the case of the light component in the selectedreference phase, unadjusted direction P_(H). The light components of thesymbol therefore appear in the display to be aligned during thatrelative movement of the display and the assumed fixed direction of gazeof the user.

Conveniently, in this example embodiment of the present invention, thetracker system may include a prediction capability so that the rotationmatrix [HW] may be synchronised to the time at which the light componentin the selected reference phase, in this example the light component inthe second phase (Pulse 2 in the example of FIG. 1) in a sequence ofthree light pulses, is expected to be visible at the display. Thisprovides for a more accurate alignment of the symbol with the intendedline of sight. However, a lack of synchronisation would not in itself beexpected to affect the achievement of a perceived alignment of the lightcomponents of the symbol. More importantly, if the rate data aresynchronised to the expected time of displaying at least one of thelight components, for example those during the selected reference phase,then it may be expected that the light components will appear moreaccurately aligned in the display, in particular where the rate ofrelative movement happens to be changing rapidly at that time.

For the light pulse in the selected reference phase, no adjustment isrequired to the matrix [HW] and equation (1) applies in positioning thesymbol in the display. However, the adjustment rotation matrix [H′H] fora first phase light component may be approximated to

$\quad\begin{bmatrix}1 & {\partial A_{z}} & {- {\partial A_{y}}} \\{- {\partial A_{z}}} & 1 & {\partial A_{x}} \\{\partial A_{y}} & {- {\partial A_{x}}} & 1\end{bmatrix}$where ∂_(A) _(x) =Δt₁₂ r_(x), ∂_(A) _(y) =Δt₁₂ y and ∂_(A) _(z) =Δt₁₂r_(z). Δt₁₂ is the time between the light pulses of the first and secondphases and r_(x), r_(y) and r_(z) are the HMD rotation rates in radiansper second resolved along the axes of the HMD frame of reference.

Similarly, and assuming Δt₃₂=−Δt₁₂, i.e. the pulse interval between thefirst and second phase light components and the pulse interval betweenthird and second phase light components are equal in magnitude butnegated, then [H′H] for the pulses of the third phase light component issimply the transpose of [H′H] for the first phase light component.

If the first phase were to be selected as the reference phase, then theadjustment rotation matrices [H′H] would be determined based upon thetimings of pulses in the subsequent second and third phases relative tothose of the first phase.

Typically, head rotations comprise only horizontal and verticalcomponents, given the design of pilot helmets and the confines of anaircraft cockpit. In such circumstances a simplification to the abovecalculations is possible whereby the roll rate r_(x) is assumed to bezero. Such a simplification provides an opportunity to adopt a simplerdisplacement method for determining an adjusted position in the displayfor space-stabilised symbols rather than the rotational adjustmentmethod described above.

The displacements required in horizontal position and vertical positionmay be determined on a similar basis to the rotational adjustment methodabove using the relative timing of pulses for the different lightcomponents and the azimuth and elevation rates of change in HMDorientation r_(y) and r_(z) respectively received from the trackersystem, according to the equations:

$\begin{matrix}{{{Shift}\mspace{14mu}{in}\mspace{14mu}{horizontal}\mspace{14mu}{position}} = \frac{s \times \Delta\; t \times r_{z}}{\cos^{2}\left( {Sym}_{x} \right)}} & (4) \\{{{Shift}\mspace{14mu}{in}\mspace{14mu}{vertical}\mspace{14mu}{position}} = \frac{s \times \Delta\; t \times r_{y}}{\cos^{2}\left( {Sym}_{y} \right)}} & (5)\end{matrix}$where

s is the angular size in radians of the centre pixel of the display,

Δt is the required time difference in seconds of the correction, inpractice the time difference between a reference time point to which thetracker system is synchronised, e.g. the time of displaying the secondlight component, as above, and the time of displaying pulses of each ofthe other components,

r_(y) and r_(z) are the HMD rotation rates in radians per secondobtained from the corrected gyro rates in HMD axes, and

Sym_(x) and Sym_(y) are the pre-shifted x and y display positions of thesymbol calculated at STEP 15.

It is assumed in equations (4) and (5) that the image projection in theHMD is a flat surface projection where, for example, the horizontaldisplay position of a symbol at an azimuth angle A_(z) is proportionalto tan(A_(z)). If the image is to be projected by spherical projection,then a simplification of equations (4) and (5) is possible in which thedivisor cos² terms may be assumed to take the value 1, such that:Shift in horizontal position=s×Δt×r _(z)  (6)Shift in vertical position=s×Δt×r _(y)  (7)

An improvement to the process described earlier with reference to FIG. 2will now be described with reference to FIG. 3, implementing thetechniques described above for positioning light components ofspace-stabilised symbols in an HMD according to an example embodiment ofthe present invention.

Referring to FIG. 3, the improved process begins at STEP 50 in respectof a given image refresh period in determining the orientation of theHMD using a known tracker system, for example in the form of a rotationmatrix [HW] as described above. At STEP 55, the rate of change inorientation of the HMD is also determined, either by the HMD trackersystem of more locally using a recent history of received orientationdata from the tracker system.

At STEP 60, the determined rate from STEP 55 is used, together withpredetermined information read from a STORE 65 giving the relativetiming of light components in the first phase relative to those in thesecond phase, to determine a rotational adjustment to the HMDorientation, for example a rotation matrix [H′H] as described above,applicable to the first phase components of space-stabilised symbols tobe displayed. Similarly, at STEP 70, the determined rate from STEP 55 isused, together with predetermined information read from a STORE 65giving the relative timing of light components in the third phaserelative to those of the second phase, to determine a rotationaladjustment applicable to the third phase components of space-stabilisedsymbols to be displayed. The determined rotational adjustments [H′H] areapplied at STEP 75 and STEP 80 to the HMD orientation [HW] determined atSTEP 50 for the first and third phase components, respectively.

The adjusted HMD orientation determined at STEP 75 for the first phasecomponents is then used at STEP 85, together with already determineddata defining a known direction to a point in inertial space, read froma STORE 90, to which a space stabilised symbol is to be aligned, tocalculate a position on the HMD image area for displaying the firstphase component of the symbol. Similarly, at STEP 95, the adjusted HMDorientation determined at STEP 80 for the third phase components isused, together with the already determined data defining the directionto the feature in inertial space, read from a STORE 90, to calculate aposition on the HMD image area for displaying the third phase componentof the symbol.

The HMD orientation determined at STEP 50 is used at STEP 100 tocalculate a position for displaying the ‘reference’ second phase lightcomponent of the space-stabilised symbol. The positions determined atSTEP 85, STEP 95 and STEP 100 are then used at STEP 105, STEP 110 andSTEP 115 respectively to draw the first, third and second phasecomponents of the symbol, as in STEP 20 of the known process shown inFIG. 2. The data defining the characteristics of pixels involved indisplaying the space-stabilised symbol are stored in an image STORE 120.

The position of any symbols to be drawn at a fixed position in the imagearea of the HMD is calculated at STEP 125 and the light components ofthe fixed symbols are drawn at STEP 130. The resultant data are storedin the image STORE 120.

Having accumulated all the data required to define pixel characteristicsfor an image refresh period, the contents of the image STORE 120 areread and used at STEP 135 to generate video data to be used, at STEP140, to generate an image at a display device of the HMD. The processthe resumes at STEP 50 for the next image refresh period.

The effects of implementing the improved image generation processdescribed above will now be illustrated with reference to FIG. 4.

Referring to FIG. 4a , and additionally to FIG. 1a , the results ofapplying the above techniques to determine the display positions for thethree phases of light components can be seen for an example rate ofrelative movement. In this example, the determined position of each ofthe pulses of the first and third phase light components intended forPixel 2 of a space-stabilised symbol can be seen to be displaced to theright (150) and to the left (155), respectively, of the second‘reference’ phase light component, as compared with the positioningshown in FIG. 1a where there is substantially no relative movement. Itcan be seen in FIG. 4a that the position of the selected ‘reference’second phase light component on the display is not changed in comparisonto its indicated position in FIG. 1a by the present invention. Ofcourse, in general, the first and third light components will bedisplaced according to the rotational correction determined at STEP 60and STEP 70, respectively, resulting in a two-dimensional vectordisplacement of those components across the image area of the HMDunless, as in this illustrated example, the direction of relativemovement of the HMD and eye gaze direction is substantially in azimuth.

Referring now to FIG. 4b , and additionally to FIG. 1b , the viewer'sperception of the effect of the repositioned light components for Pixel2 of a space-stabilised symbol is illustrated during relative movementof the HMD and eye gaze direction. As can be seen, the viewer perceivesthe light Pulses 1, 2 and 3 of the three phases of light components tobe co-located in the image area of the HMD, although slightly displacedoverall by an extent 160 from the intended position of Pixel 2, and theviewer perceives Pixel 2 of the required brightness. The fractionalshift 160 in perceived position of a displayed pixel, shown in FIG. 4bas a displacement 160, during such relative movement has been found notto impair the viewer's perception of the symbol and the overall imagesignificantly.

Example embodiments of the present invention may be implemented in anyhead or helmet-based digital display system with access to tracker datagiving an indication of eye movement relative to the display. An examplehelmet-mounted display system in which embodiments of the presentinvention may be implemented will now be described in outline withreference to FIG. 5.

Referring to FIG. 5, there is shown a representation of an HMD system,for example for use by a pilot 205 wearing a helmet 210 equipped withcomponents of a helmet tracker system and incorporating a helmet-mounteddisplay, in this example a substantially transparent waveguide display215 positioned in front of an eye 220 of the pilot 205. The trackersystem may include one or more inertial sensors 225 mounted upon thehelmet 210, arranged to supply data to a Tracker System Processor 230.The tracker system may also include an optical helmet tracker comprisingan arrangement of light-emitting diodes (LEDs) 235 integrated within theshell of the helmet 210 and controllable by the Tracker System Processor230 to emit pulses of light. The optical helmet tracker also includes anarrangement of one or more cameras 240 (one of which is shown in FIG. 5)at known fixed positions arranged to detect light from thehelmet-mounted LEDs 235 and to send corresponding signals to the TrackerSystem Processor 230.

The Tracker System Processor 230 is arranged to interpret the datareceived from the inertial sensors 225 and from the cameras 240 of theoptical helmet tracker system to determine orientation of the helmet 210and hence of the display 215 in inertial space or relative to anaircraft for example (not shown in FIG. 5) in which the pilot 205 may betravelling. The Tracker System Processor 230 may also determine a rateof change in orientation of the helmet and hence of the display 215 fromthose data inputs and output display orientation and rate of change datato an Image Generator 245. The Image Generator 245 is arranged togenerate images, including space-stabilised images for display to thepilot viewing the helmet-mounted display 215 such that they appearoverlain on the pilot's view through the transparent waveguide 215 ofthe outside world.

The Image Generator 245 may be arranged to implement the techniquesdescribed above for taking account of detected movement of the pilot'seye 220 or direction of gaze relative to the image area of thehelmet-mounted display 215, using data from the Tracker System Processor230, when positioning different light components of space-stabilisedsymbols with the aim of reducing the perceived incidence of imagesmearing or multiple image effects during movement of the eye 220 or thepilot's direction of gaze relative to the image area of the display 215.

Example embodiments of the present invention have been described in thecontext of a head or helmet-mounted digital display system as may beused on an aircraft, for example. However, the principles of operationof an improved image processor incorporating the techniques describedabove may be applied more widely, as would be apparent to a person ofordinary skill in the relevant art, to compensate for relative movementof a viewer's eye and the image area of a display and so avoid orsubstantially reduce the perceived effects of image smearing ingenerated images. Information defining such eye movement may be suppliedby a head or helmet tracker or an eye tracker for tracking the actualdisplacements of a viewer's eye, and therefore their line of sight,relative to the image area of the display. Either source of trackingdata may be used in the example embodiments of the present inventiondescribed above.

The invention claimed is:
 1. A method for controlling a display device to generate an image for viewing by a user on a display, the image being formed by at least three light components having predetermined relative timings, the method comprising: receiving image data defining an intended position of first, second, and third light components of a feature to be sequentially displayed as an element in the image; receiving rate data indicative of a rate of change in orientation of the display relative to a direction of gaze of an eye of the user; determining a displaced position on the display for displaying each of the first and third light components of the feature by adjusting the intended position of the first and third light components according to the received rate data and the predetermined relative timings relative to a timing of the second light component, the displaced position of the first and third light components being different from the intended position of the second light component; and outputting the displaced position of each of the first and third light components to the display device for display of the first and third light components at the respective displaced position, and outputting the intended position of the second light component to the display device for display of the second light component at the intended position.
 2. The method according to claim 1, wherein determining the displaced position on the display for displaying each of the first and third light components includes receiving data indicative of an orientation of the display in inertial space and determining a position on the display such that the feature appears aligned to a predetermined line of sight to a point in inertial space.
 3. The method according to claim 2, wherein adjusting the intended position of the first and third light components includes determining a respective adjustment to the received data indicative of an orientation of the display such that the displaced position on the display for displaying each of the first and third light components is determined after applying the respective adjustment to the received orientation data.
 4. The method according to claim 2, wherein the received rate data measure a rate of change in orientation of the display in inertial space and the measured rate is indicative of the rate of relative movement of the display relative to the direction of gaze of the user assuming that the direction of gaze of the user is substantially fixed in inertial space.
 5. The method according to claim 4, wherein the display is a head or helmet-mounted display (HMD) arranged to display collimated images and wherein the rate data are received from, or derived from output by, a head or helmet tracker system.
 6. The method according to claim 2, wherein the received rate data define a rate of change in orientation of the display in azimuth and in elevation resolved in a frame of reference of the display, assuming no change in orientation of the display about a roll axis in the frame of reference of the display and assuming that the direction of gaze of the user is substantially fixed in inertial space.
 7. The method according to claim 6, wherein adjusting the intended position of the first and third light components includes determining a linear displacement from a position at which the respective light component would be displayed if there was no relative movement of the display and the direction of gaze of the user, including a displacement in azimuth across the display determined using the received rate of change in orientation in azimuth combined with a displacement in elevation across the display determined using the received rate of change in orientation in elevation to give a net linear displacement across the display for display of the light component.
 8. The method according to claim 1, wherein the first, second, and third light components include light components sequentially displayed in three respective phases, the phases having said predetermined relative timings.
 9. An image processor programmed to implement the method according to claim
 1. 10. An apparatus for controlling a display device to generate an image for viewing by a user on a display, the image being formed by at least three light components having predetermined relative timings, the apparatus comprising: an input for receiving image data defining an intended position of first, second, and third light components of a feature to be sequentially displayed as an element in the image; an input for receiving rate data indicative of a rate of change in orientation of the display relative to a direction of gaze of an eye of the user; and an image processor arranged to determine a displaced position on the display for displaying each of the first and third light components of the feature by adjusting the intended position of the first and third light components according to the received rate data and the predetermined relative timings relative to a timing of the second light component, the displaced position of the first and third light components being different from the intended position of the second light component, and output the displaced position of each of the first and third light components to the display device for display of the first and third light components at the respective displaced position, and outputting the intended position of the second light component to the display device for display of the second light component at the intended position.
 11. The apparatus according to claim 10, wherein the image processor is configured to determine the displaced position on the display for displaying the first and third light components by receiving data indicative of an orientation of the display in inertial space and determining a position on the display such that the feature appears aligned to a predetermined line of sight to a point in inertial space.
 12. The apparatus according to claim 11, wherein the image processor is configured to adjust the intended position of the first and third light components by determining a respective adjustment to the received data indicative of an orientation of the display such that the displaced position on the display for displaying each of the first and third light components is determined after applying the respective adjustment to the received orientation data.
 13. The apparatus according to claim 11, wherein the received rate data measure a rate of change in orientation of the display in inertial space and the measured rate is indicative of the rate of relative movement of the display relative to the direction of gaze of the user assuming that the direction of gaze of the user is substantially fixed in inertial space.
 14. The apparatus according to claim 11, wherein the received rate data define a rate of change in orientation of the display in azimuth and in elevation resolved in a frame of reference of the display, assuming no change in orientation of the display about a roll axis in the frame of reference of the display and assuming that the direction of gaze of the user is substantially fixed in inertial space.
 15. The apparatus according to claim 10, wherein the at least three light components include light components sequentially displayed in three respective phases, the phases having said predetermined relative timings.
 16. The apparatus according to claim 10, wherein the received rate data are received from, or determined from output by, an eye tracker system associated with the display.
 17. The apparatus according to claim 10, wherein the display is a head or helmet-mounted display (HMD) arranged to display collimated images and wherein the rate data are received from, or derived from output by, a head or helmet tracker system.
 18. A computer program product comprising one or more non-transitory computer-readable mediums having stored thereon computer program code which when executed by one or more processors cause a process to be carried out for controlling a display device to generate an image for viewing by a user on a display, the image being formed by at least three light components having predetermined relative timings, the process comprising: receiving image data defining an intended position of first, second, and third light components of a feature to be sequentially displayed as an element in the image; receiving rate data indicative of a rate of change in orientation of the display relative to a direction of gaze of an eye of the user; determining a displaced position on the display for displaying each of the first and third light components of the feature by adjusting the intended position of the first and third light components according to the received rate data and the predetermined relative timings relative to a timing of the second light component, the displaced position of the first and third light components being different from the intended position of the second light component; and outputting the displaced position of each of the first and third light components to the display device for display of the first and third light components at the respective displaced position, and outputting the intended position of the second light component to the display device for display of the second light component at the intended position.
 19. The computer program product according to claim 18, wherein determining the displaced position on the display for displaying each of the first and third light components includes receiving data indicative of an orientation of the display in inertial space and determining a position on the display such that the feature appears aligned to a predetermined line of sight to a point in inertial space.
 20. The computer program product according to claim 18, wherein the at least three light components include light components sequentially displayed in three respective phases, the phases having said predetermined relative timings. 